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Summary 

We  present  here  a  research  plan  leading  to  the  specification 
of  a  network  front  end  (NFE)  which  will  meet  World  Wide  Military  Command 
and  Control  System  (WWMCCS)  needs  through  the  1980' s.   We  briefly  review 
the  current  state  of  the  art  as  it  affects  NFE  development,  identify 
some  promising  directions  for  research,  describe  tools  for  performing 
the  research,  and  present  a  detailed  plan  for  conducting  the  research. 
State  of  the  Art 

The  NFE  must  be  modular,  efficient,  and  multi-level  secure  if 
it  is  to  meet  WWMCCS  needs.   Three  groups  of  systems  are  considered 
relevant  to  NFE  development: 

existing  network  access  systems, 

secure  systems,  and 

high-bandwidth  communications  systems. 
Examination  of  these  three  groups  reveals  that  there  does  not  exist,  nor 
will  there  exist  in  the  near  future,  a  system  which  will  meet  WWMCCS 
needs . 
Research  Directions 

We  present  some  promising  ideas  for  research  which  might  lead 
to  solutions  to  NFE  problems.   Two  alternative  hardware  architectures 
are  presented  for  solving  the  bandwidth  problem.   An  alternative  soft- 
ware architecture,  the  Hub  System,  is  presented  which  may  solve  the 
problems  of  producing  a  modular,  efficient,  and  multi-level  secure 
system. 


Tools  for  Analysis  and  Design 

Carrying  out  the  research  will  require  a  number  of  basic  aids 
to  system  analysis  and  design.   These  aids  include  measurement  tools  and 
modeling  tools.  We  present  a  brief  description  of  tools  of  these  kinds 
and  how  they  are  expected  to  fit  into  the  overall  research  approach. 
Research  Plan 

We  present  a  research  plan  with  four  phases: 

I.  the  preparation  phase, 

II.  the  research  phase, 

III.  the  prototype  phase,  and 

IV.  the  specification  phase. 

The  preparation  phase  will  produce  a  set  of  technical  constraints 
for  the  design  of  the  WWMCCS  NFE  and  will  select  a  set  of  design  features 
to  be  studied  in  the  research  phase.   The  selection  will  be  performed 
through  mathematical  modeling  using  the  technical  constraints.   The 
research  phase  will  design  and  construct  a  Research  NFE  that  will  be 
used  for  evaluating  architectural  concepts  through  an  iterative  process 
of  implementation,  testing,  and  measurement.   The  prototype  phase  will 
design  and  construct  a  Prototype  NFE  which  will  serve  as  the  basis  for 
specifying  the  WWMCCS  NFE.   The  specification  phase  will  develop  the 
WWMCCS  NFE  specification. 


State  of  the  Art 

In  this  section  we  discuss  the  state  of  the  art  in  hardware 
and  software  architecture  as  it  affects  the  development  of  an  NFE.  We 
establish  a  basis  for  the  discussion  of  architectures  in  order  to 
facilitate  the  discussion  of  existing  network  access  systems,  secure 
systems,  and  existing  high-bandwidth  communications  systems.   We  evaluate 
each  of  these  systems  with  respect  to  their  potential  for  use  as  the 
NFE. 
Basis  for  Discussion 

As  a  basis  for  discussing  existing  and  proposed  architectures, 
we  present  a  set  of  system  requirements,  a  set  of  architectural  concepts, 
and  a  reference  system.   The  system  requirements  provide  a  basis  for 
evaluating  an  architecture's  ability  to  perform  adequately,  both  qualita- 
tively and  quantitatively.   The  architectural  concepts  provide  a  basis 
for  describing  system  structures.   The  reference  system  provides  a  basis 
for  comparing  architectures. 

System  requirements.   The  system  requirements  presented  here 
are  drawn  from  the  CCTC-WAD  Network  Front  End  Development  Plan  [15]. 
They  are  to  serve  solely  as  a  basis  for  evaluation  of  NFE  architectures 
and  are  not  intended  as  a  functional  description  of  a  production  NFE. 

The  system  requirements  can  be  divided  into  two  groups:  system 
characteristics  and  system  functions. 

System  characteristics  describe  overall  properties  of  a  system. 
These  may  be  further  divided  into  qualitative  characteristics  and  quanti- 
tative characteristics. 


NFE: 


The  qualitative  characteristics  of  the  NFE  are: 

1.  modular  structure, 

2.  multi-level  security,  and 

3.  reliable  operation. 

The  quantitative  characteristics  of  the  NFE  are: 

1.  support  of  250  terminals,  and 

2.  maintenance  of  a  large  fraction  of  the  theoretical 
bandwidth  on  the  host  and  network  links. 

System  functions  are  the  specific  functions  performed  by  the 


1.  communication  with  a  host  or  hosts  via  a 
host-to-front-end  protocol, 

2.  communication  with  a  network  or  networks  via 
the  appropriate  network  protocols, 

3.  communication  with  terminals  connected  both  directly 
and  via  the  telephone  network,  and 

4.  allowing  the  following  interconnection  routes: 

a.  terminal  to  host, 

b.  terminal  to  network  via  the  virtual  terminal 
protocol  appropriate  to  the  network,  and 

c.  programs  in  the  host  to  the  network  via  appropriate 
network  protocols. 

Architectural  concepts.   The  basic  categories  of  concepts 
applicable  to  NFE  software  architecture  are: 

1.  module  implementations, 

2.  module  interconnection  structures, 

3.  event  notification  mechanisms,  and 

4.  device  independence. 

Of  these,  only  module  implementations  and  event  notification  mechanisms 
are  examined  for  the  systems  considered  in  this  document. 


Module  implementation  refers  to  the  software  structure  of 
modules,  their  place  in  the  system,  and  the  means  whereby  they  are 
invoked.   The  kinds  of  module  implementation  considered  here  are: 

1.  interrupt  service  routines, 

2.  procedures, 

3.  processes,  and 

4.  isolated  processes. 

Interrupt  service  routines  (ISR's)  are  software  modules 
Initiated  by  the  hardware  whenever  certain  events  occur,  such  as  I/O 
completion  on  a  particular  device. 

Procedures  are  subroutines  called  by  other  modules. 

Processes  are  independently  executing  programs  whose  state  can 
be  saved  by  an  external  agent  at  any  time  and  restored  at  a  later  time 
without  interfering  with  the  process's  function.   A  process's  address 
space  may  or  may  not  be  isolated  from  the  address  space  of  other  processes 

Isolated  processes  are  processes  whose  address  spaces  are 
isolated  from  each  other. 

Module  interconnection  structures  refers  to  the  patterns  of 
interconnection  between  modules.   Interconnection  structures  are 

1.  static  or  dynamic,  and 

2.  multi-  or  single-level. 

Static  or  dynamic  interconnection  structure  describes  whether 
the  interconnection  structure  can  be  changed  without  terminating  the 
execution  of  a  module  and  restarting  it. 

Multi-  or  single-level  interconnection  structure  describes 
whether  the  depth  of  the  connection  structure  can  be  greater  than  one. 


Event  notification  mechanisms  are  the  means  whereby  a  module 
is  notified  that  an  event  relevant  to  it  has  occurred.   Those  considered 
here  are 

1.  blocking, 

2.  polling, 

3.  software  interrupts,  and 

4.  event  queues. 

Blocking  is  applicable  only  to  modules  which  are  processes. 
The  process's  execution  is  suspended  until  the  awaited  event  occurs. 

Polling  refers  to  the  repeated  testing  by  the  module  of  a  bit, 
called  an  event  flag,  associated  with  the  event.   The  event  flag  is  set 
by  hardware  or  by  another  module. 

Software  interrupts  associate  a  procedure  within  the  module 
with  an  event.   When  the  event  occurs,  the  procedure  is  called. 

Event  queues  associate  a  queue  with  each  module.   When  an 
event  occurs,  a  short  message,  called  an  event  notice,  is  put  in  the 
queue  of  the  affected  module.   When  a  module  is  ready  to  process  an 
event,  it  attempts  to  get  an  event  notice  from  its  queue.   If  there  is 
an  event  notice  in  the  queue,  it  is  removed  from  the  queue  and  processed. 
If  there  is  no  event  notice  in  the  queue  the  module  waits.   This  tech- 
nique is  usually  used  with  modules  implemented  as  processes  which  block 
when  their  event  queues  are  empty. 

Device  independence  refers  to  the  level  of  abstraction  at 
which  I/O  devices  are  addressed  in  the  system.   If  devices  are  addressed 
at  a  low  level  of  abstraction,  the  device  access  methods  will  be  device 
dependent  and  thus  heterogeneous.   If  devices  are  addressed  at  a  high 
level  of  abstraction,  the  access  methods  will  be  device  independent  and 


thus  homogeneous.   Device  independence  is  an  important  determinant  of  a 
system's  flexibility. 

Reference  system.   For  the  purpose  of  comparing  the  existing 
and  proposed  architectures  that  implement  the  NFE  system  functions,  a 
reference  system  is  presented  here. 

Figure  1  is  a  block  diagram  of  the  NFE  functions  in  their 
interconnection  as  modules.   The  Host  Link,  Network  Link,  and  Terminal 
Driver  modules  interface  the  NFE  to  the  host,  network  and  terminals, 
respectively.   The  Host-to-Front-End  Protocol  (HFP)  module  performs  the 
necessary  translation  between  host  and  NFE  representations  of  data  and 
control  information.   The  Virtual  Terminal  module  performs  the  necessary 
translation  between  terminals  and  host  or  network  representations  of 
data  and  control  information.   Similarly,  the  Network  End-to-End  module 
translates  between  network  and  host  or  terminal  representations  of  data 
and  control  information. 

Figure  2  shows  the  interconnection  of  NFE  modules  to  form  the 
routes  through  the  NFE.   These  routes  are  from  terminal  to  host,  from 
terminal  to  network,  and  from  host  to  network  both  directly  and  via  high 
level  (e.g.,  virtual  terminal)  protocols. 

In  order  to  simplify  the  comparison  of  existing  and  proposed 
NFE  architectures,  we  single  out  from  figure  2  the  terminal  to  network 
route.   This  terminal  to  network  route  provides  a  reference  system  for 
our  comparisons  which  is  simple  yet  still  entails  all  of  the  architectural 
features  of  the  systems  to  be  discussed.   This  reference  system  route 
with  several  channels  through  it  is  shown  in  figure  3. 
Existing  Network  Access  Systems 

The  performance  of  the  existing  network  access  systems  has 
never  been  measured  scientifically.   However,  some  characterization  of 
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performance  is  possible,  based  on  experience,  and  is  given  for  each  of 
the  systems  discussed. 

ANTS  I.   ANTS  I  (ARPA  Network  Terminal  System)  was  produced  at 
the  University  of  Illinois  in  1972  to  provide  virtual  terminal  (Telnet) 
connections  to  the  ARPA  Network  for  directly  connected  and  dial-up 
terminals.   ANTS  I  also  provides  remote  job  entry  service  to  the  UCLA 
IBM  360/91  using  a  card  reader  and  line  printer. 

The  reference  system  (figure  3)  is  implemented  in  ANTS  I  as 
shown  in  figure  4.   The  assignment  of  reference  system  functions  to 
ANTS  I  modules  is  as  follows: 

Terminal  Driver  and  Virtual  Terminal: 

Terminal  Input  ISR  (Interrupt  Service  Routine) 
Terminal  Output  ISR 
Terminal  Output  Procedure 

Network  End-to-End: 

NCP  Input  Process 
NCP  Output  Procedure 

Network  Link: 

IMP  Input  ISR 
IMP  Output  ISR 

In  the  ANTS  I  system,  all  processing  directly  associated  with 
I/O  devices,  and  even  some  network  protocol  processing,  takes  place  in 
ISR*s.   Procedures  are  used  to  interface  the  ISR's  to  one  another  and  to 
the  processes  that  perform  the  higher  level  functions.   These  procedures 
activate  the  output  ISR's  when  necessary  by  simulating  hardware  interrupts 
in  software. 

Processes  are  employed  wherever  a  function  does  not  deal 
directly  with  hardware.   These  processes  are  driven  by  event  queues.   A 
process  blocks  whenever  it  attempts  to  remove  data  from  an  empty  queue. 
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ANTS  I  supported  more  than  20  terminals  simultaneously  with 
remote  job  entry  input  and  output.   No  degradation  in  performance  was 
noticeable,  even  under  the  highest  load  to  which  ANTS  I  was  subjected. 
The  upper  limit  of  ANTS  I  performance  has  never  been  determined. 
ANTS  I  was  not  designed  with  security  in  mind.   Furthermore,  the  com- 
plexity of  its  architecture  precludes  its  being  made  into  a  secure 
system. 

ANTS  II.  ANTS  II  [1]  was  produced  at  the  University  of 
Illinois  Center  for  Advanced  Computation  in  1974  to  provide  a  wider 
range  of  functions  and  a  more  flexible  structure  than  ANTS  I. 

ANTS  II  implemented  the  reference  system  as  shown  in  figure  5. 
The  assignment  of  reference  system  functions  to  ANTS  II  modules  is 
as  follows: 

Terminal  Driver: 

Terminal  Output  ISR 
Terminal  Input  ISR 
Terminal  Procedure 

Virtual  Terminal: 

Telnet  Process 

'  Network  End-to-End: 

NCP  Process 

Network  Link: 

IMP  Input  ISR 
IMP  Output  ISR 
IMP  Input  Process 
IMP  Output  Process 

Functions  in  the  ANTS  II  system  are  performed  almost  entirely 
in  processes  which  communicate  with  each  other  via  event  queues  and 
event  notices.  A  minimum  of  processing  is  performed  by  ISR's,  which 
communicate  with  the  processes  either  via  the  processes'  event  queues  or 
by  blocking  the  processes. 
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ANTS  II  was  unable  to  support  more  than  a  few  terminals  without 
running  out  of  buffer  space  on  a  PDP-11  with  28K  words  of  memory.   ANTS  II 
was  not  designed  with  security  in  mind.   Even  though  the  ANTS  II  architec- 
ture is  simpler  than  that  of  ANTS  I,  it  is  not  clear  how  ANTS  II  could 
be  made  into  a  secure  system. 

ELF.   The  ELF  systems  were  produced  at  the  Speech  Communications 
Research  Laboratory  of  the  University  of  California  at  Santa  Barbara. 
ELF  I  was  similar  in  design  to  ANTS  I,  and  ELF  II  [8,9]  was  produced  to 
provide  a  wider  range  of  functions  and  a  more  flexible  structure.   This 
section  discusses  ELF  II.  ELF  II  was  originally  intended  to  support  research 
in  packet  speech  transmission,  but  the  design  is  general  enough  for  its 
use  as  a  general  purpose  system  for  accessing  networks. 

ELF  II  implements  the  reference  system  as  shown  in  figure  6.   The 
assignment  of  reference  system  functions  to  ELF  II  modules  is  as  follows: 

Terminal  Driver: 

Terminal  Output  ISR 
Terminal  Output  Procedure 
Terminal  Input  ISR 
Terminal  Input  Procedure 
Character  IOX  Process 

Virtual  Terminal: 

Telnet  Process 
Network  End-to-End: 

NCP  Process 

Network  Link: 

IMP  Input  Process 
IMP  Output  Process 
Block  IOX  Process 
IMP  Input  Procedure 
IMP  Input  ISR 
IMP  Output  Procedure 
IMP  Output  ISR 
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In  addition,  two  modules  perform  a  connective  function: 

EXEC  Process  (1  per  channel) 
Port  IOX  Process 

Functions  in  the  ELF  II  system  which  deal  directly  with  I/O 
hardware  are  performed  by  procedures  and  ISR's  under  the  control  of 
dedicated  IOX  (I/O  Auxiliary)  processes.   These  modules  communicate  via 
procedure  calls  and  via  event  notices.   All  other  functions  are  performed 
by  processes  which  communicate  via  event  notices. 

The  performance  of  ELF  I  was  similar  to  that  of  ANTS  I,  sup- 
porting many  terminals  with  no  noticeable  degradation  of  performance. 

The  performance  of  ELF  II  was  significantly  better  than  that 
of  ANTS  II.   On  a  PDP-11/45  with  64K  words  of  memory,  ELF  II  can  support 
32  terminals.   However,  analysis  of  its  processor  and  memory  utilization 
indicates  that  it  is  unlikely  that  ELF  II  could  meet  the  NFE  requirements. 

Neither  ELF  I  nor  ELF  II  was  designed  with  security  in  mind. 
As  with  ANTS  I  and  ANTS  II,  it  is  uncertain  whether  or  how  either  ELF  I 
or  ELF  II  could  be  made  into  a  secure  system. 

Network  Unix.  Network  Unix  [3,6]  was  produced  at  the  University 
of  Illinois  Center  for  Advanced  Computation  (CAC)  in  1975  to  provide  a 
small  general  purpose  system  with  network  access  capability.  Unix  was 
produced  by  Bell  Telephone  Laboratories  as  an  operating  system  for  the 
larger  models  of  the  Digital  Equipment  Corporation  PDP-11  series  mini- 
computers. CAC  added  ARPANET  software  and  hardware  facilities,  including 
NCP  and  TELNET. 

The  reference  system  is  implemented  in  Network  Unix  as  shown 
in  figure  7.   The  assignment  of  the  reference  system  functions  to  Network 
Unix  modules  is  as  follows: 


17 


& 


3 

a.  ec 

G   C/i 


I 


3 

O. 

ft.    id   OS 

£   3  cn 

Hd     ©     I-1 


OB 

gg 
3  o 

ft.      O.    O 

X    C    u 
l-l  (-1  ft. 


en 

CD 
01 

u 

ft.    o 

u    t- 
SE  ft- 


CO 

hum 

wan 

Z    c   O 

■J  «   o 

H   3    Id 

_ 

H  O  ft. 

i 


V 

iH  Id 

2    iJ   -O 
■H    3    01 

Cft  U 
4J  O 
01  3  U 
H  O  ft. 


I 


I 


01 

>-t  u 

§  -3 

r4  U     V 

S  3     U 

u  a.  c 

D  C     U 

H  >-•   ft. 


■*    3 
0)    3  tfl 

h6m 


I 


5  a.  a: 
<u  c  00 
H  t-t  i-i 


C 

OJ 
E 
CU 

C 

E 


e 
— 

CO 
C/5 

<u 

o 

e 

91 
U 
ID 

'*- 
0) 

as 


3 

o 

eg  — t  .-i 

3  iH  U. 

CU    (0 

3   U  C 

o-  O 

OC    u  -> 

C    3  <0 

•H   T3  B 

o   u  o 

O    O  «-. 

-4     I.  C 

«  ft.  I-I 


CU 

t- 
3 
OC 


oa  ft. 


o 

»  ■• 

4J  >. 

z  -^ 


4 


Terminal  Driver: 

Terminal  Input  ISR 
Terminal  Input  Procedure 
Terminal  Output  ISR 
Terminal  Output  Procedure 

Virtual  Terminal: 

Telnet  Input  Process  (1  per  channel) 
Telnet  Output  Process  (1  per  channel) 

Network  End-to-End: 

NCP  Procedure 
NCP  Process 

Network  Link: 

IMP  Input  Process 
IMP  Input  ISR 
IMP  Output  ISR 

Functions  in  the  Network  Unix  system  which  deal  directly  with 
I/O  hardware  are  performed  by  ISR's  and  procedures  which  communicate  via 
procedure  calls.   High-level  functions  such  as  protocol  interpretation 
are  performed  by  processes.   The  processes  communicate  with  the  procedures 
via  procedure  calls  and  blocking. 

Network  Unix  performance  noticeably  degrades  under  loads  of 
between  10  and  20  terminals.   Thus  it  cannot  meet  the  NFE  requirements. 
Short  of  a  complete  redesign,  the  internal  structure  of  Network  Unix 
precludes  its  being  made  into  a  secure  system. 

ENFE.   The  ENFE  (Experimental  Network  Front  End)  [7,10]  is 
being  produced  by  the  University  of  Illinois  Center  for  Advanced 
Computation  under  its  contract  (DCA100-76-C-0088)  with  CCTC-WAD.   It  is 
intended  to  serve  as  a  front  end  between  a  WWMCCS  H6000,  the  ARPANET, 
and  terminals.   The  ENFE  will  consist  of  Network  Unix  with  the  addition 
of  a  Host-to-Front-End  protocol  implementation  and  a  new  inter-process 
communication  (IPC)  mechanism. 
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The  reference  system  will  be  implemented  in  the  ENFE  as  shown 
in  figure  8.   The  difference  between  the  reference  system  as  implemented 
in  Network  Unix  and  as  implemented  in  the  ENFE  is  that  the  two  Telnet 
processes  required  in  Network  Unix  for  each  Telnet  connection  will  be 
combined  into  a  single  process  in  the  ENFE  for  each  Telnet  connection. 
This  is  because  the  new  IPC  mechanism  permits  a  process  to  have  multiple 
incomplete  I/O  operations  pending  simultaneously. 

The  performance  of  the  ENFE  should  be  rather  better  than  that 
of  Network  Unix,  upon  which  it  is  based.   This  expected  improvement  is 
due  to  the  reduction  in  the  number  of  processes  involved  and  to  the  use 
of  the  faster  PDP-11/70.   The  ENFE,  being  UNIX  based,  suffers  the  same 
drawbacks  vis-a-vis  security  as  does  Network  Unix. 
Secure  Systems 

The  state  of  the  art  with  respect  to  multi-level  security  as 
implemented  in  computer  systems  is  embryonic.   Producing  an  NFE  which 
is  securable  and  which  meets  the  other  NFE  requirements  will  be  a  diffi- 
cult task.   Two  relevant  efforts  are  described  below. 

Secure  Unix.   The  Unix  operating  system,  which  is  the  basis  of 
both  Network  Unix  and  the  ENFE,  currently  runs  on  DEC  PDP-11  systems. 
Two  separate  efforts  are  under  way  to  produce  a  multi-level  secure 
operating  system  whose  interface  is  as  nearly  identical  to  that  of  Unix 
as  possible.   Because  these  two  efforts,  one  at  UCLA  [11,12]  and  the 
other  at  the  MITRE  Corporation  [13],  use  the  same  basic  approach,  they 
will  be  discussed  together. 

The  basic  approach  is  to  collect,  into  a  "security  kernel," 
all  of  the  operating  system  functions  that  involve  access  to  data  by 
processes.   The  security  kernel  is  designed  and  implemented  in  such  a 
way  that  its  correct  operation  with  regard  to  multi-level  security  can 
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be  mathematically  verified.   In  order  to  make  the  security  kernel  small 
enough  to  make  this  verification  possible,  its  functions  include  only 
those  which  affect  the  security  of  data.   Operating  system  functions 
such  as  process  scheduling  are  not  included.   Thus,  in  order  to  provide 
all  operating  system  functions,  another  "layer"  of  system  is  necessary 
to  implement  the  functions  not  implemented  in  the  security  kernel.   This 
layer  contains  the  interface  to  user-level  programs  and  the  means  to 
translate  user-level  program  requests  into  requests  to  the  security 
kernel  when  required. 

When  Secure  Unix  is  operational,  the  ENFE  software  is  to  be 
merged  with  it  to  produce  a  Secure  ENFE.   Because  of  the  added  layer  of 
system  software,  we  can  only  expect  that  the  performance  of  the  Secure 
Unix  NFE  will  be  below  that  of  the  ENFE. 

SFEP.   The  Secure  Front  End  Processor  [4,14]  is  a  secure 
communications  processor  being  developed  by  the  Honeywell  Corporation. 
Its  hardware  base  is  a  Honeywell  Level  6  minicomputer  with  the  addition 
of  a  Security  Protection  Module  (SPM).   The  SPM  controls  access  to 
program  and  data  in  units  called  segments.   The  reading,  writing,  and 
execution  of  each  segment  is  separately  controllable.   Because  a  large 
number  of  segments  is  addressable  at  any  instant,  very  fine  control 
over  data  and  program  access  is  possible.   The  SPM  also  implements  a 
concept  called  "rings  of  privilege,"  which  defines  a  set  of  domains  in 
which  software  modules  execute.  Modules  executing  in  privileged  rings 
may  be  allowed  to  access  segments  which  modules  executing  in  less  privi- 
leged rings  cannot  access.   Invocation  of  privileged  modules  by  less 
privileged  modules  is  tightly  controlled.   I/O  hardware  is  also  under 
the  control  of  the  SPM.   These  features  provide  substantial  support  for 
the  implementation  of  a  secure  system. 
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At  this  writing,  it  appears  that  Honeywell  is  planning  to 
adapt  Secure  Unix  as  the  software  for  the  SFEP.   Due  to  the  additional 
support  provided  by  the  hardware,  we  can  expect  that  the  performance  of 
an  SFEP  with  a  software  system  based  upon  the  Secure  Unix  NFE  will  be 
above  that  of  the  PDP-11  based  Secure  Unix  NFE. 
High-Bandwidth  Communications  Systems 

The  Collins  C-System.   The  C-System  Model  8562  was  produced  by 
the  Collins  Radio  Group  of  Rockwell  International  and  was  first  delivered 
in  March  1974.   It  is  considered  here  because  it  is  a  high-bandwidth 
communications  processor  which  can  handle  1024  lines  simultaneously.   It 
has  been  used  as  a  front  end  to  the  SABRE  2  airline  reservation  system. 

The  C-system's  high  performance  is  a  result  of  the  architecture 
of  its  hardware  and  software.   A  simplified  diagram  of  the  C-system  is 
shown  in  figure  9. 

In  the  C-System,  very  little  processor  power  is  required  to 
handle  I/O.   The  overhead  due  to  the  state-saving  associated  with  I/O 
interrupts  is  eliminated  through  the  use  of  very  sophisticated  I/O 
hardware.   The  C-System  I/O  hardware  operates  on  a  direct  memory  access 
basis.   When  the  C-System  I/O  hardware  has  completed  its  processing  of 
a  buffer,  it  begins  to  process  the  next  buffer  automatically  and  with- 
out interruption,  and  places  an  event  notice  in  an  event  queue.   This 
event  queue  is  the  means  whereby  the  software  running  in  the  C-System 
processor  is  notified  of  I/O-related  events. 

The  C-System  software  is  organized  into  tasks  which  are  divided 
into  priority  levels  from  most  to  least  time-critical.   Tasks  are  switched 
only  when  they  are  completed  or  when  a  timer  interrupt  occurs.   When  a 
timer  interrupt  occurs,  the  state  of  the  current  priority  level  is  saved, 
and  the  most  time-critical  task  is  initiated.   Since  timer  interrupts 
occur  relatively  seldom,  this  scheme  greatly  reduces  the  overhead  due  to 

task  switching. 
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It  is  not  clear  how  the  C-System  could  be  made  secure. 

Pluribus.   The  Pluribus  system  [5]  was  produced  by  Bolt, 
Beranek,  and  Newman  Inc.  as  a  high-bandwidth  reliable  communications 
processor  for  implementing  packet  switching  nodes.   It  employs  a  multi- 
processor, multi-bus  hardware  architecture.   This  multi-processor 
architecture  provides  high  bandwidth  through  the  concurrent  processing 
of  tasks.   It  provides  reliability  because  of  the  redundancy  of  components. 
A  simplified  diagram  of  the  Pluribus  system  hardware  is  shown  in  figure  10, 

A  serious  problem, with  multi-processor  architectures  is  that 
the  allocation  of  tasks  to  processors  and  the  prevention  of  interference 
between  them  may  introduce  so  much  overhead  that  much  of  the  potential 
advantage  of  multiple  processors  is  lost.   The  Pluribus  system  solution 
to  this  problem  is  based  on  a  few  key  ideas. 

1.  The  functions  to  be  performed  by  the  system  are  broken  up 
into  very  short  tasks.   These  tasks  are  so  short  that 
each  can  run  to  completion  without  preventing  the  perfor- 
mance of  time-critical  tasks. 

2.  All  of  the  processors  are  identical.  Thus  any  free 
processor  can  undertake  any  task,  and  the  loss  of  a 
processor  will  result  in  only  a  small  loss  in  throughput. 

3.  The  tasks  to  be  processed  are  kept  in  an  event  queue 
which  is  ordered  by  priority.   Each  processor  consults 
the  queue  every  time  it  completes  a  task  and  performs  the 
highest-priority  pending  task.   Thus  the  processing 
overhead  associated  with  interrupts  is  eliminated. 

4.  The  event  queue  is  managed  by  a  special  hardware  device 
called  the  Pseudo  Interrupt  Device.  This  device  speeds 
up  queue  access  and  reduces  inter-processor  interference. 

It  is  not  clear  how  the  Pluribus  could  be  made  secure. 
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Evaluation 

Each  of  the  systems  described  above  has  some  of  the  characteris- 
tics desired  for  the  NFE.   But  none  of  the  systems  has  all  of  these 
characteristics.   The  brief  discussion  which  follows  is  summarized  in 
table  1. 

Both  ANTS  I  and  ELF  I  were  attempts  to  provide  network  access 
in  the  shortest  possible  time.   It  is  remarkable  that  both  were  highly 
efficient  in  their  use  of  machine  resources  and  thus  satisfactorily 
performed  the  functions  that  they  implemented.   But  neither  system 
employed  a  truly  modular  structure.   And  neither  was  designed  with 
security  in  mind. 

ANTS  II  and  ELF  II  were  attempts  to  rectify  the  lack  of 
modular  design  of  their  respective  predecessors.   As  such,  both  were 
successful.   But  in  both  cases  this  was  at  the  expense  of  efficiency. 
Neither  can  be  considered  a  high-bandwidth  system.   And  neither  was 
designed  with  security  in  mind. 

Network  Unix  has  the  modular  structure  to  be  expected  of  a 
time-sharing  operating  system  with  a  device- independent  I/O  subsystem. 
But  the  process-switching  orientation  of  such  systems  is  ill-suited,  and 
thus  inefficient,  for  high-bandwidth  data  communications  applications. 
The  ENFE,  which  reduces  the  number  of  processes  through  its  new  IPC 
mechanism,  is  some  improvement  in  this  respect.   But  the  improvement  is 
insufficient  to  meet  the  quantitative  requirements  for  the  NFE.   And 
Unix  was  not  designed  with  security  in  mind. 

Secure  Unix  should  solve  the  problem  of  security.   But  the 
replacement  of  a  one-layer  operating  system  with  a  two-layer  one,  which 
is  inherent  in  the  security  kernel  approach,  can  only  result  in  lower 
effective  bandwidth. 


27 


Modularity 

Efficiency 

Security 

ANTS  I 

- 

+ 

- 

ELF  I 

- 

+ 

- 

ANTS  II 

+ 

- 

- 

ELF  II 

+ 

- 

- 

Network  UNIX 

+ 

- 

- 

ENFE 

+ 

- 

- 

Secure  UNIX 

+ 

- 

+ 

SFEP 

+ 

1 

+ 

Pluribus 

1 

+ 

? 

Collins 

0 

+ 

- 

Table  1 


Evaluation  of  Systems  Considered 


Key :   +   Good 

0   Indifferent 

Bad 
?   Not  yet  determined 
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The  Honeywell  SFEP  provides  an  adequate  hardware  base  for  the 
production  of  a  secure  NFE.   If  a  suitable  software  architecture  is 
employed,  the  SFEP  might  be  able  to  meet  the  performance  goals  for  the 
NFE.   If  the  Secure  Unix  approach  is  taken,  there  may  be  some  perfor- 
mance improvement  over  the  PDP-11  implementation.   But  it  is  doubtful 
that  this  improvement  will  be  sufficient  to  overcome  the  handicaps  of 
the  security  kernel  approach. 

The  Collins  C-System  and  the  Pluribus  are  both  capable  of 
meeting  the  performance  goals  for  the  NFE.   The  Pluribus  approach  is 
especially  attractive  because  of  its  high  reliability  and  the  ability  to 
tailor  configurations  to  the  required  bandwidth.   Both  of  these  charac- 
teristics are  due  to  the  Pluribus'  multi-processor  architecture.   It  is 
not  clear  how  either  system  could  be  made  secure. 
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Research  Directions 

The  state  of  the  art  and  its  history  shows  that  the  production 
of  an  NFE  with  all  of  the  desired  characteristics  is  a  difficult  task. 
However,  the  achievements  to  date  suggest  directions  for  research  which 
can  potentially  solve  the  problems  involved.   In  this  section  we  present 
some  promising  ideas  for  research  in  hardware  and  software  architecture 
relevant  to  the  NFE. 

We  discuss  two  alternative  hardware  architectures.  The  first 
is  an  inexpensive  multi-processor.  The  second  is  an  efficient  CPU  which 
operates  without  interrupts. 

We  then  present  an  alternative  architecture  for  a  communications 
software  system  called  the  Hub  System.   This  system  design  has  some 
promise  as  a  base  for  an  NFE  which  meets  all  of  the  requirements. 
Alternative  Hardware  Architectures 

Multi-microprocessors  with  common  memory.   The  advent  of 
powerful  microprocessors  combined  with  the  development  of  high-speed  IC 
memories  suggests  that  an  extremely  high-bandwidth  communications  processor 
could  be  constructed  very  inexpensively.   For  example,  the  Texas  Instruments 
TMS9900  microprocessor  [16]  is  a  single  chip  16-bit  CPU  that  has  the 
same  instruction  set  and  highly  efficient  context  switch  mechanism  as 
the  TI  990  mini-computer.  Typical  instruction  execution  times  for  the 
TMS9900  are  on  the  order  of  5  microseconds.   IC  memories  are  now  available 
with  cycle  times  on  the  order  of  50  nanoseconds.   Thus  the  memories  are 
two  orders  of  magnitude  faster  than  the  microprocessors.  This  suggests 
that  a  large  number  of  microprocessors  could  access  a  common  memory 
without  contention  and  with  a  fairly  simple  and  inexpensive  interface, 
producing  a  system  with  an  extremely  high  effective  processing  rate. 
This  possibility  certainly  bears  further  study. 
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The  Bunch  interruptless  system.  Context  saving  and  restoring 
due  to  interrupts  is  a  major  source  of  overhead  in  any  communications 
processor  based  on  traditional  hardware  architecture.   The  Collins  C- 
System  and  the  Pluribus  both  solve  this  problem  by  implementing  task 
queues  in  hardware  which  are  interrogated  by  the  CPU(s).  Another 
solution,  proposed  by  Bunch  [2],  involves  structuring  the  CPU  so  that 
contexts  for  a  number  of  processes  are  supported  completely  by  CPU 
hardware.  While  the  CPU  still  has  a  single  ALU  and  instruction  fetch- 
decode-execute  mechanism,  there  is  a  complete  set  of  registers,  including 
the  program  counter,  for  each  high  priority  process.   Each  of  these 
processes  is  responsible  for  a  hardware  I/O  management  task.   The  processes 
are  organized  according  to  a  priority  system.  A  "ready  bit"  is  associated 
with  each  priority  level.  When  an  I/O  device  requires  attention,  it 
turns  on  the  ready  bit  of  its  assigned  priority  level.   If  this  priority 
level  is  higher  than  that  of  the  currently  running  process,  the  CPU 
switches  register  sets  when  the  current  instruction  execution  is  finished. 
This  operation  should  only  take  a  few  tens  of  nanoseconds.  When  a 
process  has  completed  its  task,  it  causes  its  ready  bit  to  be  shut  off, 
and  the  CPU  switches  to  the  register  set  of  the  highest  priority  process 
whose  ready  bit  is  set.  This  scheme  permits  the  use  of  a  process- 
switching  architecture  without  its  attendant  overhead. 
An  Alternative  Software  Architecture:   The  Hub  System 

The  Hub  System  is  a  communications  software  design  concept 
developed  at  the  University  of  Illinois  Center  for  Advanced  Computation 
over  the  past  two  years.  The  Hub  System  has  promise  as  a  way  to  minimize 
the  trade-off  between  efficiency  on  the  one  hand  and  modularity  and 
security  on  the  other. 

The  Hub  System  would  implement  the  reference  system  as  shown 
in  figure  11.  The  assignment  of  reference  system  functions  to  Hub  System 
Modules  is  as  follows: 
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Terminal  Driver: 

Terminal  Input  ISR 
Terminal  Output  ISR 
Terminal  Procedure 

Virtual  Terminal: 

Telnet  Procedure 
Network  End-to-End: 

NCP  Procedure 

Network  Link: 

IMP  Procedure 
IMP  Input  ISR 
IMP  Output  ISR 

With  the  exception  of  the  ISRfs,  the  modules  of  the  Hub  System  correspond 
one-to-one  with  the  functions  of  the  reference  system.   Furthermore, 
only  one  copy  of  each  module  is  required  even  though  there  may  be  many 
channels.   None  of  the  Hub  System  Modules  is  implemented  as  a  process; 
all  are  implemented  as  sets  of  procedures.  All  such  procedures  are 
called  by  one  single  process  -  the  HUB.   This  architecture  eliminates 
the  considerable  overhead  involved  in  process  implementation  and  process 
switching.   In  this  way,  the  Hub  System  may  be  able  to  meet  the  performance 
requirements  for  the  NFE  (e.g.,  support  of  250  terminals). 

The  elimination  of  the  overhead  associated  with  processes  is 
made  possible  by  the  way  in  which  Hub  System  Modules  communicate  with 
one  another.  All  communication  between  Hub  System  Modules  goes  through 
the  Hub  Queue.   To  communicate  with  another  Hub  System  Module,  a  given 
Hub  System  Module  calls  a  system  procedure.   The  system  procedure  enters 
a  request  in  the  Hub  Queue,  and  then  returns.   The  entry  in  the  Hub 
Queue  is  processed  in  its  turn  by  the  HUB,  which  calls  the  appropriate 
Hub  System  Module  procedure.   This  sequence  is  illustrated  in  figure  12. 
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Every  Hub  System  Module  consists  of  a  set  of  procedures,  each 
of  which  corresponds  to  a  type  of  event,  e.g.,  "dispatch":   the  generation 
of  a  message  to  be  transmitted.  For  example,  when  an  event  occurs  in 
the  Telnet  Hub  System  Module  that  is  relevant  to  the  NCP  Hub  System 
Module,  the  Telnet  module  calls  the  system  procedure  corresponding  to 
the  type  of  event.   The  system  procedure  then  enters  an  event  notice  of 
the  corresponding  type  into  the  Hub  Queue.  When  the  HUB  removes  the 
event  notice  from  the  Hub  Queue,  it  calls  the  corresponding  procedure 
from  the  set  of  procedures  comprising  the  NCP  module.  When  that  procedure 
has  completed  its  processing  of  the  event,  it  returns  control  to  the 
HUB.   This  sequence  is  illustrated  in  figure  13. 

The  Hub  System  turns  on  the  HUB.   Its  operation  is  that  of  a 

cyclical  process: 

step  n    The  HUB  waits  until  there  is  an  event  notice 
in  the  Hub  Queue. 

step  n+1  The  HUB  takes  the  first  event  notice  from 
the  Hub  Queue. 

step  n+2  The  HUB  calls  the  appropriate  procedure  in 
the  appropriate  Hub  System  Module. 

step  n+3  The  called  procedure  performs  its  task 
(which  may  entail  the  calling  of  system 
procedures  to  put  new  event  notices  in  the 
Hub  Queue,  etc.)* 

step  n+4  The  called  procedure  returns  control  to 
the  HUB. 

step  n+5   (same  as  step  n,  next  cycle  begins) . 


* 


Note  that  the  Hub  Queue  mechanism  implements  a  recursive 
process  iteratively. 
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The  structure  of  a  Hub  System  Module  as  a  set  of  procedures 
that  are  invoked  one  at  a  time  requires  that  state  information  be  saved 
by  each  procedure  between  procedure  invocations.   This  information  must 
be  kept  separately  for  each  channel  using  the  module.   To  each  channel 
using  a  module  there  is  assigned  an  area  of  memory  called  a  Channel 
Memory.   Thus  there  is  a  set  of  Channel  Memories  associated  with  each 
Hub  System  Module.   By  this  means,  multiple  copies  of  the  same  module 
are  rendered  unnecessary.  Also  associated  with  each  channel  using  a 
module  is  a  set  of  ports  called  Channel  Ports.   These  ports  are  used  to 
define  the  route  taken  by  the  channel,  i.e.,  the  interconnections 
between  modules.  Together,  a  Channel  Memory  and  its  associated  Channel 
Ports  define  a  Channel  Stage.   This  is  illustrated  in  figure  14. 

A  Hub  System  which  implements  all  of  the  NFE  functions  is 
shown  in  figure  15. 

The  Hub  System  design  is  efficient.   There  is  no  process 
switching.  The  inter-module  communication  processing  overhead  is  very 
small.   The  tables  required  to  keep  track  of  procedure  sets,  Channel 
Memories,  and  Channel  Ports  are  small.   The  Channel  Memories  themselves 
contain  only  the  information  necessary  for  modules  to  perform  their 
functions . 

The  Hub  System  design  is  inherently  modular.   Inter-module 
communication  is  completely  standardized.  Modules  are  necessarily 
broken  down  into  procedures  which  handle  well-defined  events. 

Most  important,  the  Hub  System  design  is  securable.  All 
inter-module  communication  is  performed  via  system  procedures.   These 
system  procedures  are  so  simple  that  an  integrated,  verified,  single- 
level  secure  system  may  be  practical.   The  Channel  Ports  for  each  stage 
define  the  connections  between  stages.  This  provides  an  efficient 
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mechanism  for  controlling  the  flow  of  data  between  stages.  Given  an 
appropriate  hardware  base,  such  as  the  Honeywell  Level  6  with  its  Security 
Protection  Module,  it  may  be  possible  to  construct  a  high-bandwidth 
secure  NFE  using  the  Hub  design. 
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Tools  for  Analysis  and  Design 

A  number  of  basic  tools  will  be  needed  in  the  research  project 
for  system  analysis  and  design.   These  tools  fall  into  two  categories: 

1.  measurement  tools,  to  study  the  performance  of  any  front  end 
that  is  built,  and  to  identify  design  features  needing 
improvement;  and 

2.  modeling,  to  analyze  front-end  features  and  design  details 
before  implementation. 

In  this  section  we  briefly  describe  the  tools  of  these  kinds  that  will 

be  useful  for  studying  alternative  front-end  architectures.   We  also 

indicate  how  these  tools  are  expected  to  fit  into  the  overall  research 

approach. 

Measurement  Tools 

Tools  that  are  needed  for  studying  the  performance  of  an 
implemented  computer  system  fall  into  two  categories.   First,  there  is 
the  software  that  accumulates  and  analyzes  data  on  the  events  which 
occur  within  the  system.   Second,  there  is  software  and/or  hardware  to 
exercise  the  system  in  a  controlled  manner.   For  a  front  end,  the  latter 
will  consist  mainly  of  message  generators. 

Much  of  the  event  data  to  be  collected  will  include  the  time 
when  the  event  occurred.   A  crystal-controlled  programmable  clock  must 
be  available  to  provide  accurate  timings.   By  time-stamping  messages  at 
various  points  as  they  proceed  through  the  system,  it  is  possible  to 
tell  how  long  messages  spend  being  processed  by  various  modules,  waiting 
in  various  queues,  etc.   It  will  probably  be  helpful  to  be  able  to 
examine  complete  histories  for  individual  messages,  as  well  as  to  collect 
statistics  (means,  variances,  etc.)  on  message  timings.   Probes  must  be 
embedded  in  the  front-end  software  to  attach  time-stamps  to  messages  at 
appropriate  points. 
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As  an  alternative  to  time- stamping  messages,  an  event  trace 
mechanism  could  be  built.   Such  a  mechanism  allows  arbitrary  data 
relevant  to  an  event  (e.g.,  event  type  and  time  of  occurrence)  to  be 
stored  in  an  event  trace  buffer  whenever  specified  events  occur.   This 
data  can  then  be  analyzed  and/or  read  out,  at  the  experimenter's  con- 
venience.  Types  of  events  which  are  of  potential  interest  include: 

1.  the  arrival  of  a  message  at  a  queue, 

2.  the  pickup  of  a  message  from  a  queue, 

3.  the  filling  of  a  buffer  or  queue,  and 

4.  the  breakdown  of  interprocess  communication  because  of  the 
lack  of  message  space. 

Naturally,  the  full  set  of  "interesting"  events  is  highly  dependent  upon 

the  frontrrend  architecture. 

A  monitoring  facility,  such  as  is  provided  by  Unix,  will  also 
be  of  use.  A  monitor  works  by  examining  the  program  counter  at  specified 
time  intervals.   The  monitor  maintains  counts  of  how  often  the  system  is 
discovered  to  be  executing  the  various  processes.   In  this  way,  a 
statistical  picture  can  be  built  up  of  how  much  time  the  processes  are 
taking.   Since  observations  made  at  regular  time  intervals  are  not 
independent  and  hence  may  yield  misleading  results,  it  might  be  advan- 
tageous to  generate  the  monitoring  interrupts  at  random  intervals. 

The  message  generators  to  exercise  the  system  must  be  able  to 
produce  messages  of  varying  lengths  and  at  specified  time  intervals. 
The  lengths  could  be  fixed  as  specified  by  the  experimenter,  or  they 
could  be  randomized  -  perhaps  normally  distributed  with  a  given  mean  and 
variance.   Similarly,  the  time  intervals  between  messages  should  satisfy 
a  specified  probability  distribution  (e.g.  exponential,  if  the  sending 
of  messages  is  assumed  to  be  a  Poisson  process) . 
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If  various  paths  through  the  front  end  are  to  be  exercised 
simultaneously  the  message  generator  must  have  the  capability  of  fanning 
out  the  messages  to  provide  prescribed  loadings  on  the  various  paths. 
In  this  way  it  may  be  possible  to  study  the  impact  of  the  different 
front-end  functions  on  each  other. 

The  terminal-handling  capacity  of  the  front  end  must  also  be 
determined.   If  this  is  to  be  done  by  generating  an  artificial  terminal 
load,  one  way  to  accomplish  this  is  to  attach  an  external  computer  to 
the  terminal  interface.   Software  to  generate  pseudo  terminal  traffic 
would  then  reside  in  this  external  computer. 

Finally,  testing  of  the  front  end  in  a  realistic  environment 
requires  attaching  the  front  end  to  the  host  and  the  network  for  which 
it  was  designed.  Additional  software  and/or  hardware  may  be  required  to 
implement  realistic  end-to-end  testing. 
Modeling 

In  general,  modeling  of  a  system  involves  the  construction  of 
a  formal  description  of  the  system  -  almost  always  simplified  -  from 
which  system  behavior  may  be  deduced.   In  network  front  ends,  certain 
parts  or  .aspects  of  the  system  are  amenable  to  study  by  modeling.   For 
example,  flow  control  mechanisms  or  buffering  strategies  can  be  modeled 
and  their  impact  on  throughput  examined. 

The  model  -  or  formal  description  -  is  generally  couched  in 
mathematical  terms.   It  could  range,  depending  upon  the  application, 
from  a  simple  algebraic  equation  to  a  mathematical  construct  of  con- 
siderable complexity  and  sophistication. 

Once  one  has  a  model,  some  technique  must  be  used  to  extract 
from  the  model  the  information  desired  (for  example,  expected  system 
throughput) .   The  building  of  a  model  is  a  process  involving  a  certain 


43 


amount  of  intuition  and  judgement,  and  hence  not  readily  specified  in 
terms  of  set  rules.   Extracting  information  from  the  model  is,  however, 
a  process  subject  to  straightforward  mathematical  techniques.   In  the 
remainder  of  this  section  we  discuss  several  techniques  most  likely  to 
be  of  use  in  studying  network  front  ends. 

It  should  be  noted  that,  even  though  a  model  is  an  oversim- 
plification of  reality  and  its  validity  is  questionable  ja  priori,  the 
predictive  ability  of  a  model  can  be  (and  should  be)  checked  by  com- 
parison with  experimental  results. 

Simulation.   Computer  simulation  involves  taking  a  formal 
prescription  for  system  behavior  and  actually  mimicking  that  behavior 
in  a  step-by-step  fashion.   The  problem  is  that  under  different  condi- 
tions (i.e.,  with  different  inputs  or  initial  state)  the  system  behaves 
differently.   Thus  many  simulations  must  be  carried  out  and  analyzed 
statistically  to  build  a  complete  picture  of  system  behavior. 

Nevertheless,  simulation  has  proved  itself  to  be  an  important 
tool  in  computer  system  design.   In  testing  a  design  implementation, 
cause  and  effect  can  be  obscured  by  the  complexity  of  the  system  under 
observation.  A  simulation  study  can  help  to  interpret  ambiguous  test 
results.   Simulation  can  also  be  useful  in  fine-tuning  a  system.   It  is 
often  much  simpler  to  try  a  proposed  system  change  by  making  the  change 
in  a  system  simulator  than  by  actually  changing  the  implementation. 

General,  event-driven  simulation  systems  are  readily  available, 
Such  systems  are  generally  flexible  enough  to  allow  for  both  determinis- 
tic, scheduled  events  (e.g.,  the  arrival  of  a  previously  dispatched 
message)  and  stochastic  events  (e.g.,  the  generation  of  a  message  by  a 
process) . 
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The  major  components  of  a  typical  simulation  system  are  an 
event  generator,  a  general  queue  manipulation  routine,  and  a  set  of 
statistical  routines.   The  event  generator  is  the  heart  of  the  simula- 
tor.  Given  a  list  of  event  names,  the  associated  event  distributions 
(i.e.,  probability  distributions  for  occurrences)  and  initial  instances, 
this  routine  generates  the  corresponding  sequences  of  events.   Examples 
of  events  are  the  arrival  and  pickup  of  messages.   The  events  are 
generated  randomly  with  the  prescribed  probability  distributions.   The 
queue  manipulation  routine  handles  the  addition  and  deletion  of  events 
from  queues,  properly  maintaining  the  queue  linkages.   The  statistical 
routines  gather  data  on  events  and  compute  relevant  statistics,  as 
requested  by  the  user.   In  addition,  there  may  be  a  provision  for  the 
user  to  write  his  own  data  analysis  routines. 

Even  though  simulation  systems  of  this  kind  are  available, 
it  would  require  an  enormous  amount  of  additional  work  to  set  up  a 
realistic  simulation  of  an  entire  network  front  end.   Simulations  also 
tend  to  be  expensive,  having  to  run  for  long  periods  of  time  to  generate 
statistically  valid  results.   For  these  reasons,  it  is  unlikely  that 
full-scale  simulations  of  front-end  behavior  can  be  carried  out. 
Simulations  of  selected  features  of  the  front  end  (such  as  flow  control 
mechanisms  or  buffer  allocation  strategies)  may,  however,  help  to  identify 
places  where  changes  in  the  front-end  design  could  lead  to  performance 
improvement. 

Markov  process  analysis.   The  main  disadvantage  of  simulations  - 
namely,  the  long  run  times  required  to  generate  valid  statistics  -  can 
sometimes  be  avoided.   One  way  to  do  this  is  to  model  the  behavior  of 
interest  as  a  Markov  process,  mathematically  describable  as  a  large  set 
of  linear,  algebraic  equations,  which  may  then  be  solved  by  computer. 
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The  first  step  in  building  a  Markov  model  is  to  define  a  set 
of  states  describing  the  subsystem  to  be  modeled.   The  system  moves  from 
state  to  state  as  time  passes.   A  transition  matrix  T  can  then  be  defined 
with  elements  T   giving  the  probability  of  a  transition  from  state  j  to 
state  i  in  time  t.   These  first  steps  are  not  as  difficult  as  they  might 
appear.   Something  like  a  simulation  program  may  be  used  to  automatically 
determine  successive  possible  states  and  to  keep  track  of  the  transition 
probabilities. 

The  matrix  T  can  then  be  used  to  obtain  steady-state  popula- 
tion densities,  or  probabilities  that  the  system  is  in  a  given  state. 
Suppose  that  p  is  a  column  vector  whose  components  p .  are  the  probabil- 
ities that  the  system  is  in  state  j.   Then  the  steady-state  probabilities 

must  satisfy 

Tp  =  p. 

This  homogeneous  system  of  linear,  algebraic  equations  may  be  solved  for 
p  by  one  of  the  numerical  techniques  available  for  large,  sparse  matrices. 

The  problem  is  that,  for  a  system  of  any  complexity,  the 
number  of  states,  and  hence  the  matrix  T,  is  enormous.  We  have  used  a 
Markov  model  to  study  a  simple  flow-control  scheme  and  found  several 
hundred  states  necessary.   To  specify  a  state  for  flow-control  purposes 
requires  specifying  the  number  of  full  (or  empty)  buffers  at  each  end 
and  the  locations  (necessarily  discretized)  of  all  messages  and  acknow- 
ledgements in  the  system.   Nevertheless,  we  found  that  our  Markov  model 
readily  yielded  results  on  throughput  and  buffer  utilization.   (For 
example,  if  states  1  and  2  are  the  only  ones  for  which  all  send  buffers 
are  full,  then  p.  +  p~  gives  the  fraction  of  the  time  that  all  send 
buffers  are  expected  to  be  full.)   In  more  complicated  cases,  not  only 
will  the  computation  become  very  time-consuming,  but  also  it  may  be 
difficult  to  interpret  the  huge  array  of  population  densities  obtained. 
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One  could,  however,  build  and  use  Markov  models  for  a  thorough  study 
of  proposed  flow  control  mechanisms  both  within  the  front  end  and  between 
host  and  front  end.   Buffer  management  can  also  be  studied  by  Markov 
models.   Like  simulation,  Markov  analysis  could  also  be  very  helpful  in 
fine-tuning  these  aspects  of  the  system.   Unfortunately,  the  proliferation 
of  states  with  system  complexity  makes  it  unlikely  that  this  tool  will 
be  useful  on  a  broad  scale. 

Queueing  theory.   Queueing  theory  is  the  study  of  waiting 
lines  for  a  service,  or  set  of  services.   Jobs  are  assumed  to  enter  the 
system  and  to  join  a  queue  for  service.  After  moving  to  the  head  of  the 
queue,  they  are  processed  and  either  leave  the  system  or  join  the  queue 
for  another  service.   Some  simple  queueing  systems  may  be  analyzed  as 
Markov  processes.   For  example,  in  a  single-queue,  single-service  system, 
the  system  states  are  defined  by  a  single  variable:   the  number  of  jobs 
in  the  queue.   The  transition  matrix  then  tends  to  be  very  simple  and 
solvable  by  analytic  means. 

Queueing  theory  is  a  natural  way  to  determine  buffer  needs  at 

the  various  points  in  a  front  end.   Consider,  for  example,  the  set  of 

buffers  belonging  to  any  particular  module.   Assume  for  simplicity  that 

there  is  no  flow  control.   Suppose  that  we  can  measure  or  estimate  the 

rate  of  arrival  of  messages  to  be  processed  and  the  time  it  then  takes 

to  process  each  message.   Queueing  theory  will  immediately  yield  the 

probabilities  p  that  there  are  n  messages  waiting  to  be  processed.   We 
n 

would  then  choose  the  number  N  of  buffers  to  be  large  enough  so  that 

n>N  n 
is  very  small  (  so  that  the  buffers  seldom  overflow) .   Flow  control 
will,  of  course,  prevent  buffer  overflow.   But  the  queueing  theory 
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analysis  is  still  useful,  since  if  message  arrivals  must  be  slowed 
because  of  a  shortage  of  buffers,  system  throughput  will  decrease. 

It  is  likely  that  the  more  complex  aspects  of  a  front  end  will 
not  be  amenable  to  queueing  theory  analysis.   However,  queueing  theory 
may  be  successfully  used  to  model  the  simpler  aspects  of  front-end 
behavior. 
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Research  Plan 
Overview 

The  Alternative  Architecture  Research  Plan  will  lead  to  the 
specification  of  an  NFE  which  will  meet  WWMCCS  needs  through  the  1980 's. 
Figure  16  is  a  diagram  of  the  complete  research  plan.   The  research  plan 
consists  of  four  phases: 

1)  the  preparation  phase, 

2)  the  research  phase, 

3)  the  prototype  phase,  and 

4)  the  specification  phase. 

Each  of  these  phases  is  further  divided  into  stages. 

Preparation  phase.   The  preparation  phase  will  produce  a  set 
of  technical  constraints  for  the  design  of  the  WWMCCS  NFE  and  will 
select  a  set  of  design  features  to  be  studied  in  the  research  phase. 
The  preparation  phase  will  use: 

1)  the  existing  knowledge  of  WWMCCS  user  requirements  [18,19,20,21, 
22,23,24], 

2)  the  Experimental  Network  Front  End  (see  p. 19), 

3)  •  the  literature  on  computer  architecture,  and 

4)  the  analytical  tools  described  in  the  preceding  section. 

Research  phase.   The  research  phase  will  use  the  technical  con- 
straints and  selected  design  features  from  the  preparation  phase  to 
construct  a  Research  NFE  (RNFE)  which  will  serve  as  a  test  bed  for  arch- 
itectural concepts.   Each  design  feature  will  be  implemented  and  tested, 
insofar  as  is  possible,  in  isolation.   The  design  features  which  are 
most  promising  and  which  at  the  same  time  comprise  a  complete,  compatible, 
NFE  system  design  will  then  become  the  RNFE.   The  resulting  RNFE  may 
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then  undergo  several  iterations  of  a  cycle  consisting  of  redesign,  con- 
struction, and  evaluation.  After  each  iteration,  a  decision  point  will 
determine  whether  to: 

1)  iterate  again, 

2)  proceed,  or 

3)  terminate  the  program. 

When  sufficient  confidence  in  the  RNFE  has  been  gained,  the  RNFE  will 
be  evaluated  in  a  network  environment.   Another  decision  point  will 
determine  whether  to  iterate  as  above  or  to  proceed  into  the  prototype 
phase.   The  prototype  phase  will  use  the  set  of  technical  constraints 
produced  in  the  preparation  phase,  and  the  knowledge  gained  in  the 
research  phase,  to  design  and  implement  a  Prototype  NFE  (PNFE) .   An 
iterative  cycle  similar  to  that  employed  for  the  RNFE  will  be  employed 
for  the  PNFE. 

Specification  phase.   In  the  specification  phase,  the  PNFE 
and  the  set  of  technical  constraints  will  become  the  basis  for  the 
WWMCCS  NFE  specification. 

There  follows  a  detailed  description  of  each  of  the  phases  of 
the  research  plan. 
Preparation  Phase 

The  preparation  phase  (see  figure  17)  will  lay  the  groundwork 
for  the  rest  of  the  project.   It  has  two  major  goals: 

1)  to  identify  the  technical  constraints  on  the  design 
of  the  NFE  system,  and 

2)  to  select  design  features  appropriate  for  investigation 
in  the  research  phase. 

The  technical  constraints  will  serve  both  as  a  basis  for  selecting  design 
features  in  the  preparation  phase  and  as  a  basis  for  evaluating  progress 
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in  the  other  phases  of  the  project.   The  design  features  will  be  in- 
vestigated further  in  the  research  phase  for  possible  incorporation  in 
the  Research  NFE. 

In  the  preparation  phase  the  primary  tool  will  be  mathematical 
modeling.   This  will  allow  many  design  features  to  be  evaluated  relatively 
economically. 

The  preparation  phase  has  six  stages: 

1)  the  study  stage, 

2)  the  load  estimation  stage, 

3)  the  constraint  determination  stage, 

4)  the  gross  specification  stage, 

5)  the  screening  stage,  and 

6)  the  comparison  stage. 

Each  of  these  stages  is  described  below. 

Study  stage.   The  study  stage  will  establish  a  data  base  which 
will  influence  every  phase  of  the  project.   The  data  will  be  drawn 
from  three  sources: 

1)  the  existing  studies  of  WWMCCS  user  requirements, 

2)  the  ENFE  experiments  (see  [17]),  and 

3)  the  relevant  published  literature. 

The  existing  studies  of  WWMCCS  user  requirements  will  be  used 
to  develop  two  sets  of  WWMCCS  NFE  characteristics.   They  will  be 
instrumental  in  determining  the  technical  constraints  on  the  NFE  design. 
The  first  set  of  WWMCCS  NFE  characteristics  consists  of  the  load  mix  of 
applications  and  users  which  must  be  supported  on  the  WWMCCS  network 
under  both  normal  and  unusual  conditions.   The  load  mix  will  be  used  in 
the  load  estimation  stage.   The  second  set  of  WWMCCS  NFE  characteristics 
consists  of  system  characteristics  such  as  availability,  response  time, 
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security,  and  flexibility.  This  second  set  of  characteristics  will 
be  used  in  the  constraint  determination  stage. 

The  ENFE  experiments  will  be  used  to  develop  two  sets  of 
measurements.   They  will  be  used  in  determining  the  technical  constraints 
on  the  NFE  design.   The  first  of  these  two  sets  of  measurements  consists 
of  internal  measurements  of  the  ENFE.   These  internal  measurements  will 
facilitate  both  understanding  the  system  bottlenecks  inherent  in  the 
architecture  of  the  ENFE  and  also  identifying  the  resources  required  in 
the  NFE  regardless  of  its  architecture.   The  internal  measurements  will 
be  used  in  the  constraint  determination  stage.   The  second  of  the  two 
sets  of  measurements  consists  of  measurements  of  the  load  placed  on  the 
ENFE  by  known  applications  running  in  the  WWMCCS  H6000  under  known 
conditions.   This  set  of  load  measurements,  expressed  as  statistical 
distributions  of  message  sizes  and  message  inter-arrival  times,  will 
be  used  in  the  load  estimation  stage. 

The  literature  on  computer  hardware  and  software  architecture, 
communication  processor  design,  computer  security  and  protection,  and 
other  relevant  topics  will  be  searched  to  determine  the  state  of  the 
art  in  existing  designs  related  to  NFE  problems.   Proposed  but  untested 
designs  will  also  be  sought  out.   The  information  obtained  through  this 
literature  search  will  be  used  in  the  gross  specification  stage  and  in 
the  screening  stage. 

Load  estimation  stage.   The  load  estimation  stage  will  produce 
estimates  of  the  traffic  load  on  the  NFE  under  both  normal  and  unusual 
conditions.   The  estimates  of  traffic  load,  expressed  as  statistical 
distributions,  will  be  used  in  the  constraint  determination  stage. 
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A  mathematical  model,  the  load  model,  will  be  constructed 
using  information  produced  in  the  study  stage: 

1)  the  load  mix  derived  from  the  WWMCCS  user  requirements,  and 

2)  the  load  measurements  obtained  from  the  ENFE  experiments. 
The  load  model  will  be  used  to  obtain  estimates  of  the  traffic  load 
distribution.   The  values  assigned  to  parameters  of  the  model  will  be 
varied  to  obtain  estimates  of  the  traffic  load  on  the  NFE  under  normal 
conditions  and  also  under  unusual  conditions.   Unusual  conditions  include 
both  crisis  situations  and  loss  of  computing  or  communication  resources. 

As  an  example,  let  us  assume  that  the  load  mix  shows  that  a 
particular  NFE  in  the  WWMCCS  system  will  normally  have  100  active 
terminal  users,  75  using  application  A  on  the  local  host  and  25  using 
application  B  on  a  remote  host.   The  ENFE  load  measurements  for  each  of 
applications  A  and  B  will  be  consulted.   The  load  mix  and  the  load 
measurements  will  then  be  used  to  generate  parameter  values  for  simulations 
which  yield  a  measure,  in  bits  per  second  and  messages  per  second,  of 
the  traffic  load  on  the  NFE  under  these  normal  conditions. 

Now  let  us  assume  that  the  foreign  host,  which  normally  supports 
application  B,  becomes  inaccessible.   Let  us  further  assume  that,  under 
these  conditions,  the  local  host  must  support  application  B  as  well  as 
application  A,  and  that  there  are  100  remote  users  of  application  B. 
This  information  will  be  used  to  generate  new  parameter  values  for  simula- 
tions which  yield  a  measure  of  the  traffic  load  on  the  NFE  under  these 
unusual  conditions. 

Constraint  determination  stage.   The  constraint  determination 
stage  will  produce  a  set  of  technical  constraints.   The  NFE  design  will 
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have  to  meet  these  constraints  if  it  is  to  fulfill  the  WWMCCS  user  require- 
ments. The  technical  constraints  will  be  used  in  two  stages  of  the  prepar- 
ation phase:   the  gross  specification  stage  and  the  comparison  stage.   The 
technical  constraints  will  also  be  used  in  the  other  phases  of  the  project. 

A  mathematical  model,  the  constraint  model,  will  be  constructed 
using  the  information  produced  in  previous  stages: 

1)  the  system  characteristics  developed  in  the  study  stage 
from  the  WWMCCS  user  requirements, 

2)  the  internal  measurements  obtained  in  the  study  stage 
from  the  ENFE  experiments,  and 

3)  the  estimated  traffic  load  distribution  produced  in  the 
load  estimation  stage. 

Some  of  the  constraints  will  come  directly  from  the  WWMCCS  user  require- 
ments.  They  will  be  expressed  in  quantities  such  as: 

1)  minimum  percent  system  availability, 

2)  maximum  response  time, 

3)  maximum  probability  of  security  violation,  and 

4)  maximum  cost  of  system  modification. 

Other  constraints  will  come  from  more  sophisticated  analysis,  using 
techniques  such  as  simulation.  These  constraints  will  be  expressed 
in  quantities  such  as: 

1)  minimum  required  buffer  space, 

2)  minimum  throughput  in  messages  per  second,  and 

3)  maximum  processing  time  per  character. 

Gross  specification  stage.   The  gross  specification  stage  will 
produce  a  set  of  gross  specifications  for  the  NFE  design.   These  gross 
specifications  will  be  used  in  the  screening  stage  as  the  criteria  for 
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selecting  design  features  for  closer  examination.   The  gross  specifications 
will  be  produced  through  system  analysis  using: 

1)  the  technical  constraints  developed  in  the  constraint 
determination  stage,  and 

2)  the  state  of  the  art  in  hardware  and  software  design 

as  discovered  by  the  literature  search  performed  in  the  study 

stage. 
The  relation  between  this  information  and  the  gross  specifications 
may  be  clarified  by  the  following  examples. 

1)  The  system  availability  constraint  may  lead,  in 
the  present  state  of  the  art,  to  the  specification 

of  an  NFE  system  with  redundant  hardware;  i.e.,  multiple 
processors,  busses,  memories,  etc. 

2)  The  minimum  buffer  space  constraint  may  be  used  to 
specify  the  minimum  amount  of  memory  which  the  NFE  system 
must  be  able  to  support  and  address. 

3)  The  security  constraint  may  lead  to  a  specification 
of  the  functional  characteristics  of  the  memory  pro- 
tection hardware  for  the  NFE  system. 

4)  The  throughput  and  security  constraints  may  lead  to 

the  specification  that  certain  operating  system  functions 
be  implemented  in  hardware  or  firmware. 

Screening  stage.   The  screening  stage  will  select  hardware 
and  software  designs  which  are  to  be  compared  in  the  comparison  stage. 
The  designs  to  be  considered  will  be  those  identified  by  the  literature 
search  in  the  study  stage.   Both  existing  designs  and  proposed  designs 
will  be  examined.   The  selection  will  be  based  upon  the  compatibility  of 
each  design  with  the  gross  specifications.   For  instance: 


*7 


1)  An  operating  system  design  might  be  rejected 
because  it  cannot  support  multiple  processor 
configurations . 

2)  A  mainframe  design  might  be  rejected  because  it 
makes  memory  protection  difficult. 

Comparison  stage.   The  comparison  stage  will  compare  the 
designs  selected  by  the  screening  stage.   Using  mathematical  modeling, 
the  comparison  will  determine  the  designs  which  best  meet  the  technical 
requirements.   The  designs  which,  under  the  analysis,  best  solve  the 
problems  posed  by  the  constraints  will  be  selected  for  empirical  examination 
in  the  research  phase. 
Research  Phase 

The  research  phase  (see  figure  18)  will  evaluate  design 
features  by  implementing  them  and  then  performing  experiments  using  the 
implementations.   This  approach  is  empirical,  as  distinguished  from  the 
modeling  approach  used  in  the  preparation  phase.   The  outcome  of  the 
experiments  will  be  the  Research  NFE  (RNFE) ,  which  will  have  been  tested 
in  actual  network  use.   The  RNFE  will  serve  in  the  prototype  phase  as  a 
basis  for  the  design  of  the  Prototype  NFE. 

The  research  phase  will  use  both  the  technical  constraints  and 
the  selected  design  features  produced  in  the  preparation  phase.  It  will 
produce  both  the  RNFE  and  the  RNFE  Experiment  Plan  which  will  be  used  in 
the  prototype  phase. 

The  research  phase  will  consist  of  three  stages: 

1)  the  isolated  experiment  stage, 

2)  the  integrated  experiment  stage,  and 

3)  the  network  experiment  stage. 
Each  of  these  stages  is  described  below. 
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Isolated  experiment  stage.   The  isolated  experiment  stage 
will  further  refine,  vis-a-vis  the  technical  constraints,  the  evaluation 
of  each  of  the  design  features  selected  in  the  preparation  phase.   This 
evaluation  will  require,  for  each  design  feature: 

1)  implementing  it  to  a  point  sufficient  for  experimentation,  and 

2)  evaluating  it  via  experiment. 

The  evaluation  of  the  design  features  will  not  require  the  implementation 
of  an  entire  system  for  each  design  feature.   The  outcome  of  the  isolated 
experiment  stage  will  be  a  set  of  usable  design  features  which  will  be 
employed  in  the  integrated  experiment  stage. 

Integrated  experiment  stage.   The  integrated  experiment  stage 
will  select  a  set  of  design  features  which  can  be  used  in  designing  a 
preliminary  RNFE.   These  usable  design  features  will  be  mutually  compatible 
and  will  comprise  a  complete  design  for  a  basic  communication  processing 
system.   The  preliminary  RNFE  will  contain  a  set  of  software  and  hardware 
modules  sufficient  for  evaluating  the  integrated  design.   The  preliminary 
RNFE  need  not  contain  the  complicated  hardware  and  software  modules 
required  for  network  access.   This  restriction  may  permit  a  relatively 
economical  evaluation  of  the  preliminary  RNFE  as  a  communications  pro- 
cessor. A  plan,  the  RNFE  Experiment  Plan,  will  be  developed  for  testing 
and  evaluating  the  RNFE. 

The  integrated  experiment  stage  will  use  the  technical  con- 
straints developed  in  the  preparation  phase  both  to  guide  the  design 
of  the  preliminary  RNFE  and  to  serve  as  a  basis  for  the  design  of  the 
RNFE  Experiment  Plan. 

The  integrated  experiment  stage  will  employ  a  cyclic  process 
of  design,  construction,  and  evaluation.   The  evaluation  will  proceed 
according  to  the  RNFE  Experiment  Plan.   After  each  cycle,  a  decision 
will  be  taken  as  to  whether  to: 
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1)  redesign  part  or  all  of  the  RNFE, 

2)  reimplement  part  or  all  of  the  RNFE, 

3)  go  on  to  the  network  experiment  stage,  or 

4)  terminate  the  program. 

Network  experiment  stage.   The  network  experiment  stage  will 
be  entered  when  a  preliminary  RNFE  has  passed  evaluation.   In  this 
stage,  hardware  and  software  will  be  constructed  for  connecting  the  RNFE 
to  an  existing  network  which  mimics  the  expected  WWMCCS  network  environ- 
ment.  The  resulting  RNFE  will  then  be  subjected  to  a  rigorous  test  and 
evaluation  under  realistic  conditions.   The  evaluation  will  proceed 
according  to  the  RNFE  Experiment  Plan.   A  cyclic  process  will  again 
be  employed;  at  the  end  of  each  cycle,  it  will  be  decided  whether  to: 

1)  return  to  some  point  in  the  integrated  experiment  stage, 

2)  go  on  to  the  prototype  phase,  or 

3)  terminate  the  program. 
Prototype  Phase 

The  prototype  phase  (see  figure  19)  will  construct  a  prototype 
NFE  (PNFE),  using  the  architectural  knowledge  gained  in  the  research 
phase.   The  PNFE  will  serve  as  the  basis  for  the  specification  of  the 
WWMCCS  NFE.   A  plan,  the  PNFE  Experiment  Plan,  will  be  developed  for 
testing  and  evaluating  the  PNFE. 

The  prototype  phase  will  use  the  technical  constraints 
developed  in  the  preparation  phase,  and  the  RNFE  developed  in  the 
research  phase,  to  guide  the  design  of  the  PNFE.   The  experience  gained 
in  designing  and  using  the  RNFE  Experiment  Plan  will  be  used  in  designing 
the  PNFE  Experiment  Plan. 

The  PNFE  will  be  a  complete  WWMCCS  NFE  employing  state-of-the-art 
technology.   It  will  contain  all  of  the  hardware  and  software  necessary 
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for  connecting  WWMCCS  hosts  to  the  networks  and  devices  used  in  the 
WWMCCS  environment. 

The  completed  PNFE  will  be  subjected  to  a  cyclic  test  and 
evaluation  process  using  the  PNFE  Experiment  Plan.   At  the  end  of  each 
cycle  it  will  be  decided  whether  to: 

1)  redesign  the  PNFE, 

2)  alter  the  implementation  of  the  PNFE, 

3)  go  on  to  the  specification  phase,  or 

4)  terminate  the  program. 
Specification  Phase 

The  specification  phase  (see  figure  20)  will  produce  the 
specification  for  the  WWMCCS  NFE. 
The  specification  phase  will  base  the  specification  on: 

1)  the  technical  constraints  developed  in  the  preparation 
phase,  and 

2)  the  PNFE  developed  in  the  prototype  phase. 
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